by S. Stanley Young, John D. Dunn, and Warren Kindzierski
A new study by a team of University of Southern California researchers claims that children exposed to poor air quality are at greater risk of (self-reported) bronchitis symptoms than are adults. But this health claim is tenuous.
Published in the American Journal of Respiratory and Critical Care Medicine, the study uses data sets from a 30-year-old Southern California Children’s Health Study cohort—with a long length of time between exposure and presumed response of self-reported bronchitis.
In addition to the time delay, the study—led by professor Erika Garcia—suffers from other problems.
First, it contains a heavy crossover between asthma, which is an allergic disease, and bronchitis. Patients who claimed they suffered from bronchitis (but were treated for asthma) have seen their symptoms disappear.
Further, doctors know that inhaling irritating substances, such as industrial fumes, causes bronchitis. In addition to irritant substances, causes and other risk factors include viruses, bacteria, cigarette smoke, low immunity, and repeated bouts of severe heartburn.
The notion that self-reported bronchitis symptoms are due to air quality components alone—and air quality components from childhood—is worthless.
Second, self-reported symptoms are known to be unreliable health indicators because of “awareness bias,” meaning a bias in environmental health research. This is the tendency to report illness or symptoms because of concerns arising from proximity to a hazard in the absence of a measurable biological effect.
Self-reported symptoms can’t be considered objective unless they can be confirmed with other more credible measures, such as a physician’s diagnosis.
Third, academic researchers today have virtually infinite flexibility to manipulate study methods to get preferred results. Substantial reporting has flagged flaws in research methods because of questionable practices. Below, we discuss two questionable practices that corrupt the Garcia study, but more may exist.
Take, for instance, dredging. Scallops are harvested commercially using specialized equipment called dredges, in which a metal frame with a mesh bag is dragged along a seafloor to collect scallops. In dredging this way for scallops, other material called bycatch is unintentionally collected. Collecting this bycatch may cause harm to marine ecosystems.
Academic research has its own form of both data dredging and serious harm. This involves using questionable research practices known as fishing expeditions, phacking, cherry picking, and HARKing. All of these involve searching through data using unplanned analyses and then reporting significant findings without properly describing the methods and processes by which the findings were obtained.
The Garcia study is p-hacked by reanalyzing a subset of data when, after the researchers’ main analysis, they further examined statistical associations between prenatal exposure to poor air quality and self-reported bronchitis symptoms later in life.
One bias that corrupts environmental health observational studies, including the Garcia study, is called multiple testing and multiple modeling, or MTMM.
This bias may occur when academics use their data set to test multiple outcomes, multiple predictors, different population subgroups, multiple statistical cause-effect models, or multiple confounders to cause-effect associations. The bias increases the likelihood of making a Type I (false positive) error. This bias also is referred to in the literature as multiple comparisons or multiplicity.
The number of statistical comparisons made in a study, referred to as Search Space, may be estimated to assess the potential for multiple testing biases. These estimation procedures are well-established for observational studies of air quality and heart attacks, asthma attacks, and development of asthma, and for gas stoves and asthma attacks.
The Search Space was estimated for the Garcia study. Researchers’ main analysis performed an estimated 131,072 statistical comparisons, while their secondary analysis (data subset re-analysis) performed an additional 262,144 statistical comparisons. The total number of statistical comparisons is 393,216. The study also noted that researchers made no statistical correction for this bias.
The Garcia study is a perfect example of data dredging that is corrupted with numerous biases. Spot-checking in Google Scholar of other (almost 300) publications that used the Southern California Children’s Health Study cohort data set reveals the same multiple testing bias problems. A single published study for a particular cohort data set may address only the tip of the iceberg of statistical comparisons.
Overall, other hypotheses may be at issue, considering that the same cohort data set may be used many times for research. Many publications in the literature for a single cohort data set suggest a large number of statistical comparisons examined with possibilities for large numbers of false positive (chance) findings present overall.
The Garcia study claims that children exposed to poor air quality are at greater risk of self-reported bronchitis symptoms as adults. But the conclusion reveals a clear lack of causation between exposure and response, and this study shouldn’t be used as the basis for policy.
– – –
S. Stanley Young, Ph.D, is a visiting fellow for the Science Advisory Committee in The Heritage Foundation’s Center for Energy, Climate, and Environment. He is CEO of CGStat in Raleigh, North Carolina, and director of the National Association of Scholars’ Shifting Sands Project. Dr. John D. Dunn is a visiting fellow for the Science Advisory Committee in The Heritage Foundation’s Center for Energy, Climate, and Environment. Warren B. Kindzierski, Ph.D., now retired, is a former professor of public health at the University of Alberta, Canada.
Photo “People in Poor Air Quality” by ElizabethViera. CC BY-SA 4.0.
